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ABSTRACT 

Apprehension about evaluating the use of computers in 
schools can be attributed to a variety of reasons, including the 
claim that evaluation discourages innovation and discovery • Powerful 
reasons, however, make evaluation at both regional and national 
levels desirable. By conceiving of evaluation as a formative, 
on-going process rather tnan as a summative process, it becomes 
possible to identify problems in a project from the start, allowing 
developers to taJce corrective action. Additionally, formative 
evaluation can provide generalizable data which may facilitate the 
implementation of similar projects elsewhere, thus reducing waste of 
time and money, a^ important consideration given the public's demand 
for accountability in education. Formative evaluation contributes to 
the validation process by making possible the establishment of a 
broader base of information which can be utilized to more reliably 
chart change and growth in programs. In fact, it can be a positive, 
helpful activity rather than a negative, judgmental activity if 
provision is made for effective communication about the goals of 
evaluation with staff members; the process is open to the emergence 
of unanticipated questions and to the discussion of program 
shortcomings; an external evaluator is employed; a system of regular 
reporting to program staff is included; and system goals for the use 
of computers are periodically clarified with program staff. 
Evaluations incorporating these guidelines are unlikely to stifle the 
energy and enthusiasm that are so critical in the innovation process. 
(GL) 
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COMPUTERS IN EDUCATION: AN EXPLORATORY AREA 

There is no doubt that computers in education offer exciting 
prospects. There is also no doubt that much of what is happening with 
computers in education is still at the exploratory level. Technological 
possibilities are continually expanding at the same time that access to 
sophisticated systems and software becomes more affordable. Teachers 
are being challenged to discover new opportunities for themselves and 
their students. In fact^ the energy and enthusiasm of committed, 
visionary individuals is a powerful component of growth in the area of 
computer-related applications in education. 

SHOULD WE BE THINKING ABOUT ASSESSMENT? 

Clearly, we wish to nurture the growth and excitement that fuels 
so much of the work in computers in education. , It seems, therefore, 
that assessing or evaluating computer use in your region or country is 
not a particularly appropriate suggestion at this point in time. Some 
reasons that are given for feeling this way include: 

1. Evaluation will discourage innovation and discovery, 
in that it will apply some predetermined standard to 

a situation 

2. Evaluation makes teachers in the process of growth 
and experimentation become uncomfortable or even 
disinclined to be innovative 

3. How can we evaluate, when we are just experimenting, 
trying out new and different things? 

4. We are not "far enough" into a project, or into 
computer use in general, to be thinking about 
evaluation. We should wait for two or three years 
until everything is set up and the project is running 
smoothly 

5. Evaluation costs time and money which just aren't 
available, given all the demands involved with 
getting hardware and software in schools 

6. We don't know how to go about doing this kind of 
evaluation. We have a "student assessment" branch but 
we have no procedure for trying to assess the impact 
of an innovation 
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7. The impact that computers are making on our students 
is important but subtle; for example, we don't 
expect to see changes in their "higher-level 
thinking for many years 

8. We don't really need to "evaluate" the impact of 
computers in education; we can see that good things 
are going on, that a lot of activity is taking place, 
and that "the students love it." 



All of these are reasdable explanations. In fact, so reasonable, 
that they are the case in most districts and countries. However, I 
would like to argue that there are some powerful reasons that make 
"evaluating" desirable, here and now, in the context of computer-related 
activity in schools. After I discuss these reasons, give some examples, 
and suggest some strategies for evaluating computer-related activity at 
the regional or national level, I would like to come back to these 
reasons for not evaluating. I submit that all of them, reasonable as 
they are, can be reconsidered. 



THINKING ABOUT EVALUATION AS CONSTRUCTIVE INSIGHT INTO ONGOING 
ACTIVITIES 



Many of the reactions people have to the process of "being 
evaluated" relate to evaluation used for summative, or judgmental 
purposes. With this type of evaluation, some conclusions are offered as 
to how well an activity has been conducted and has met its goals. 
However, there are other approaches to evaluation, one of which is the 
well known "formative evaluation." Formative evaluation works within an 
ongoing activity or system, and has as a major purpose the contribution 
of insights that can be used immediately by those within the system for 
refinements or adjustments in what they are doing ♦ 

There is a way of thinking about formative evaluation which is 
particularly helpful in situations involving computer-related activity. 
This approach involves a distinction between the logic of the 
assumptions of a situation and the actual activities that take place. 
When things do not go as one hopes, on a small or large scale, it may be 
that basic assumptions about the individuals involved/ or the likelihood 
that certain actions will result in certain outcomes need to be 
reexamined. Or, it may be that it is only adjustments in the actual way 
of executing an activity that need to be made. Distinguishing between 
these two aspects can make a valuable contribution, both in an immediate 
fashion and also for longer-term planning. Let us look at some examples 
of this approach: 
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Applying Onooing Evaluation to a New Project 



Using telecommunications as an instructional tool is an exciting 
new idea in education. There are many projects throughout the world in 
which some exploration with telecommunications in the classroom is going 
on. A typical project involves a school in one country making contact 
with schools in two other countries to jointly produce a small 
newsletter. This sort of contact is expected to be good for the 
students — they will be more motivated to write because they will want to 
communicate with their new "colleagues," they will have a new and more 
effective way of learning about life in different cultures, and they 
will experience writing in a new context — as a social activity involving 
group discussions about content, editing, and clarity of expression. In 
addition, they will be developing a new set of computer-related skills, 
important because of the increasing importance of telecommunications in 
the workplace. 

The typical project with these goals, however, encounters various 
difficulties, some technical, some "human". Most frequently, these are 
technical. However, other problems reoccur: 

What if one of the schools doesn't meet its deadlines, 
or doesn't send materials at all? 



What about different times for school vacations in 
different places,? 

What about different languages? 

What if only a few students seem to be doing all the 
communicating? 

What if there is a change in teachers at one of the 
schools and the new teacher doesn't feel comfortable 
handling the telecommunications process? 



These and many other real problems have occurred at 
telecommunication projects that I have been asked to evaluate. A 
typical comment made to me in situations involving the first year of a 
project is, "Well, these are all predictable start-up problems. We will 
have the bugs worked out by next year." But unfortunately, a typical 
second-year situation is that the initial motivation for the project has 
subsided, some of the key people from the first year are no longer 
involved in the project, follow-up support and maintenance are not 
available in the way they were at the project onset because the 
starting-up of new projects is absorbing attention, and the project 
ceases to happen. 
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Very few of the many exploratory telecommunications projects going 
on throughout the world include systematic evaluation as part of the 
project from its start. Thus, variations on the pattern I have 
described above occur again and again. Very often no one is there to 
interpret what is going on, on an ongoing basis, in order to clarify a 
distinction between problems that relate to implementation decisions and 
those that relate more fundamentally to assumptions and logical 
planning. Although participants in a project are, of course, making 
continual interpretations about their problems, it is frequently helpful 
chat an outside observer (or evaluator) work with them to identify their 
own assumptions and consider which of their difficulties relate to their 
inclement at ion decisions and which to the logic of their prior 



assumptions. Also, an evaluator can be responsible for ongoing, 
interpretive written comments about the project, extremely valuable to 
those beginning other similar projects in different settings. 
Strangely, this sort of ongoing trace of decisions is rarely available, 
and thus, collectively, we repeat "exploratory" projects over and over, 
without building on each others' experiences. 

Examples of Other Situations where Ongoing Evaluation Is Useful . 

Ongoing formative evaluation can also be used to considerable 
advantage during educational software development projects. Rather than 
waiting for teacher evaluation of finished products, the evaluator can 
organize teacher reaction at various points during the 
conceptualization, design, and development phases, when feedback can 
make a more substantial impact on the product itself. Formative 
evaluation can make a helpful contribution to the delivery of 
information technology courses in schools, by providing an ongoing look 
at how such courses are actually being implemented in the classrooms. 
The distinction between plans for the courses and the impact of 
individual teachers and situations as the courses are implemented can 
provide a valuable dimension to the interpretation of the contribution 
of these courses. An evaluator can also provide a useful contribution 
through a midstream look at ongoing regional or national practice and 
policy, especially through identifying areas where comriianication of 
objectives, opportunities, and experiences is not occurring as 
effectively as those wno are organizing the communication assume it is. 

Advantag es of an External Evaluator for Ongoing Evaluation 

In each of these examples, all of which I hnve participated in as 
an external evaluator, there are particular advantages of working with 
an "informed outsider" as evaluator rather than only using individuals 
within the system. 

-I have found that participants are mors, comfortable in 
identifying their problems and experiences with 
someone who is not the designer or deliverer of those 
experiences. 

-An external evaluator can help to clarify assumptions 
operating within the system that those who are 
immersed in the system may not be aware of. 

-The external evaluator can look at execution 
strategies in a somewhat more objective manner than 
can those involved in doing the execution. He can, 
for example, see the strategies separate from the 
personal characteristics of those who implement the 
strategies, a distinction that may not be as easy to 
see in reflecting about one's own activities. 

-Goals and plans change continually in technology- 
related educational enterprises; those who are part of 
the ongoing evolution of these changes may not be 
aware of how far they have moved from the original 
intentions of an activity. The external evaluator, as 
an informed "historian", can ask questions about this 
evolution* 
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-The external evaluator^ because he arrives at certain 
times rather than always being part of the situation^ 
can serve as a catalyst to discussion and periodic 
reconsideration of goals and objectives. In the day- 
to-day implementation of overall regional or national 
practice^ hese sort of periodic consiaerations of 
goals and objectives can be quite valuable, 
particularly as such goals are frequently in the 
process of change and maturation themselves. 

-Finally, as was commented on in the context of 
telecomiTiunications projects, information synthesis 
between implementation sites with regard to computer 
use in schools does not occur well; too often we 
"reinvent the wheel", when we could benefit 
considerably in time, money, and ideas from being 
better informed about each other's experiences. A 
more wide-spread development of systematic formative 
evaluations at the regional and national level could 
make an information base about these experiences one 
step closer to being of use to individuals in other 
settings . 



THINKING ABOUT EVALUATION BECAUSE SOONER OR LATER WE WILL HAVE TO 



The example** I gave above showed contributions that can accrue to 
the region as well as to the field in general from the more systematic 
use of formative evaluation relative to computers in education. 
However, I believe there are other reasons why evaluation will more and 
more become part of the computers-in-education context, along with 
"dynamics and vision." There is increasing evidence that the period of 
public willingness to invest considerable portions of its educational 
budget on exploratory work in this area is evolving into a call for some 
kind of accountabillcy. There is, after all, only so much time and 
money available to the overall educational system. We cannot expect the 
public to continue investing this time and money without some concrete 
evidence of "better educated" students. 

So far, this evidence is not available in any generalizable form. 
Part of the reason the evidence is not available is lack of systematic 
efforts to assess Oj. evaluate, for the reasons listed earlier in this 
paper. Another part of the reason, however, is that research that has 
been done is frequently criticized as highly flawed and, taken in 
combination, yields contradictory results. 

I believe that we must, now, begin the process of developing 
better and more sensitive procedures for assessing the impact of the 
time and money we are committing to computer use in education, because 
soon society and its governments will require some evidence of this 
assecssment from us. The United States Office of Technology Assessment, 
for example, has just completed a major study of computer use in 
education in the United States. An entire section of this report (one 
of the seven chapters) is focussed on the need for cost-effectiveness 
analysis as a basis for better decision making about public funding in 
education. 
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We cannot expect that, after ten years of computers in education 
in much of the Western world, we can keep getting funds for "vision" 
and "potential" if w.e cannot show some generalizable evidence that this 
vision can be translated into a payoff with high priority relative to 
our students. Reports from individuals describing their own experiences 
are not enough; taken in isolation, each report may reflect the personal 
characteristics of the individuals in the situation as much or more than 
It reflects, the cost and time of computer involvement. Our arguments 
become much stronger, however, if we have a way to systematically 
evaluate and consolidate experiences over a broader area than the 
individual teacher, school, or project. The formative benefits of 
ongoing evaluation at the regional or national level can be augmented by 
the benefits that can come from .the availability of a broader base of 
information in order to more reliably chart change and growth than is 
now possible. 

SO, EVALUATION IS A GOOD IDEA, BUT HOW CAN WE BEGIN? 

Providing the sort of constructive evaluation described here is as 
much an art as a science. This is both good news, in that formal and 
cumbersome procedures do not have to become institutionalized before 
valuable applications of formative evaluation can start. But it is 
"bad" news, in that it does not help the decision maker know where and 
how to begin. Here are some strategies that may help: 

1. Communicate to your staff that formative, 

ongoing evaluation is a positive, helpful 
activity, not a negative, judgmental one. 
Also work to develop the attitude that it is 
a good and helpful thing to discuss 
experiences that have not gone as hoped, in 
order to learn from them. 

2. Consider an approach to ongoing evaluation 

that systematically separates considerations 
about the premises and assumptions of a 
project from observations about the specific 
execution of a project. 

3. Try to ask specific questions about the impact of 

what is being done with computers, but be open to 
the emergence of new questions. Encourage "brain- 
storming" over strategies to assess the 
questions. Sometimes just getting a good question 
is enough to better focus a staff on common 
goals. 

4. Begin by trying to identify what you are using now 

as evidence that computers are making a positive 
contribution. Sometimes this evidence will be 
only at the level of, "Look at the smiles on the 
students' faces," but that is a good start. Keep 
track of the types of evidence that now influence 
your decision making about computers, and expect 
to see these change as your experiences mature. 

5. Consider using an external evaluator who will make 

periodic visits to your situation to contribute 
different insights relative to your assumptions 
and the execution of your activities. 
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6. Disseminate the results of these reflections and 

of the work of the external evaluator in an 
informal but regular way throughout your system. 
This helps establish evaluation as a collegial 
exercise, not as a formal statement or judgment, 

7. Periodically discuss the implicit and explicit 

goals that your system has for computers in 
education. All activities, evaluation or imple- 
mentation, can develop more smoothly if staff 
share a sense of goals and vision. In our 
rapidly growing field, this vision will continual- 
ly be changing; thei-efore regular, informal, 
discussion about goals is valuable. An external 
evaluator can stimulate this discussion. 

WHAT ABOUT THE "GOOD REASONS" WE HAVE FOR NOT EVALUATING COMPUTER- 
RELATED ACTIVITY? 

At the start of this presentation, we identified aight common, and 
reasonable, reasons that decision makers in education frequently give 
for not systematically implementing ongoing evaluation with respect to 
computer use in their systems. Let us reexamine these reasons. 

The first four can be dispelled if we stop thinking of evaluation 
as summative and judgmental, and instead see it as a positive 
contribution to the better understanding of what is going on at the 
moment. The logic and assumptions of an activity can always be 
discussed, even if the assumption is no more specific than "Exploration 
is good for education" and the project is only beginning. 

The fifth point is based on an assumption that "an evaluation" 
requires an extensive enterprise, thus costing time, money, and burden 
on a system. The type of evaluation I am suggesting as most helpful for 
emerging computer use in education can be approached on a small and 
personal scale and still have value. The costs of inviting an external 
evaluator in for two- or three-day discussions, once or twice a year, 
can be more than repaid in terms of input for more effective decision 
making in the system. 

The next two points remain serious objections. We don't know how 
to "measure" the impact on computer use on our students. Maybe this 
impact will only show up many years in the future; maybe we don't have 
sophisticated enough strategies to capture what is already going on. 
However, because these are problems doesn't mean we can ignore them. We 
are making implicit assessments all the time about computer use in our 
schools, and these judgments and assessments underlie our decisions to 
spend more money and other resources on this uiie. We should, at least, 
come to a better understanding of what motivates our current decision 
making, but more than this, I believe we will eventually have to have 
more professional strategies for cost-benefit analysis in this area. 
The time to start development of these strategies is now, before we are 
forced to impose them. 

The last point on the list of reasons to postpone systematic 
evaluation efforts was the assumption that we really don't need 
evaluation in the area of computers in education at all, that we "know" 
it is good and that good things are happening. I share this belief that 
"good things" are happening; however, I want to make those "good things" 



more widely available to other students. I think that this will occur 
more efficiently and effectively than it does now if we analyze our 
vision so that we can better identify things that can thwart its 
momentum. Yes^ there are many good things going on with computers in 
schools, but I believe that much more could be happening, given the 
resources that are already available. I think ongoing, positive, 
formative evaluation can be h "productivity tool" and I ^rge you to 
consider including it now in your systems. The co-relation of ongoing 
evaluation and "dynamics and vision" can be one of mutual value. 
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